User Interface for Impact Analysis

ABSTRACT

Data is received characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs. A first coefficient characterizing an interaction between a first input and a second input of the plurality of inputs is calculated based on a first set of weights of the plurality of weights. The first set of weights are indicative of interaction between the first input and the second input. A graphical object including a plurality of regions is displayed in a graphical user interface (GUI). A first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively. A second graphical object connecting the first region to the second region is displayed in the GUI. Related apparatus, systems, techniques and articles are also described.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of and priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63/223,829 filed Jul. 20, 2021, the entire contents of which is hereby expressly incorporated by reference herein.

TECHNICAL FIELD

The subject matter described herein relates to a user interface for improved impact analysis and representation.

BACKGROUND

A Graphical User interface (GUI) which can be displayed on a display device (e.g., monitor) of a computing device can allow a user to interact with the computing device. The GUI can include interactive graphical objects. Actions in the GUI can be performed through direct interaction (e.g., clicking, double clicking, etc.) with the interactive graphical objects. An interaction with the interactive graphical object can result in execution of a software application. In some implementations, the results of the software application can be displayed in the GUI.

SUMMARY

In an aspect, data is received characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs. A first coefficient characterizing an interaction between a first input and a second input of the plurality of inputs is calculated based on a first set of weights of the plurality of weights. The first set of weights are indicative of interaction between the first input and the second input. A graphical object including a plurality of regions is displayed in a graphical user interface (GUI). A first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively. A second graphical object connecting the first region to the second region is displayed in the GUI. A first visual characteristic of the second graphical object is indicative of the first coefficient.

One or more of the following features can be included in any feasible combination. For example, the graphical object can depict a circle. The first region and the second region can be a first portion of the circumference of the circle and a second portion of the circumference of the circle. The second graphical object can depict a chord connecting the first portion of the circumference of the circle to the second portion of the circumference of the circle. The first visual characteristic can be a thickness of the chord. The impact function can include a sum of a plurality of terms, and each term of the plurality of terms can be a product of a weight of the plurality of weights and at least one input of the first plurality of inputs. Calculating the first coefficient can be based on a first set of terms of the plurality of terms, and each term of the first set of terms can be a product of the first input, the second input, and a weight of the first set of weights. The first interaction coefficient can include the sum of the first set of terms. Data characterizing an analytical task and a target output of the analytical task can be received. The first set of weights can be calculated by the analytical task and based on the first plurality of inputs. An impact score can be calculated by the impact function. The the impact score can be indicative of a prediction of achieving the target output when inputs of the analytical task are the first plurality of inputs.

Non-transitory computer program products (i.e., physically embodied computer program products) are also described that store instructions, which when executed by one or more data processors of one or more computing systems, causes at least one data processor to perform operations herein. Similarly, computer systems are also described that may include one or more data processors and memory coupled to the one or more data processors. The memory may temporarily or permanently store instructions that cause at least one processor to perform one or more of the operations described herein. In addition, methods can be implemented by one or more data processors either within a single computing system or distributed among two or more computing systems. Such computing systems can be connected and can exchange data and/or commands or other instructions or the like via one or more connections, including a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc.

The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims.

DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart of an exemplary method of generating an interface representative of a multi-step analytical process;

FIG. 2 illustrates an exemplary graphical user interface (GUI) display space that includes a first node;

FIG. 3 illustrates the GUI display space of FIG. 2 that includes a first set of nodes that are displayed adjacent to the first node;

FIG. 4 illustrates the GUI display space of FIG. 3 that includes a second set of nodes that are displayed adjacent to a node selected from the first set of nodes;

FIG. 5 illustrates the GUI display space of FIG. 4 that includes a third set of nodes that are displayed adjacent to another node selected from the second set of nodes;

FIG. 6 is an image of an example graphical user interface according to some example implementations of the current subject matter that can provide an intuitive interface to managing and performing analytical tasks using a model building and deployment platform;

FIG. 7 is another example of the summary view interface as described with respect to FIG. 6 ;

FIG. 8-10 illustrate an example interface for building a predictive model using the example expandable node approach, for example, a model building view;

FIG. 11-12 is another example interface illustrating data transformation and data preparation views;

FIG. 13A-13C illustrates an example of data transformation;

FIG. 14 illustrates an interface for selecting different projects or creating a new project;

FIG. 15A-15B illustrates an interface describing instructions for a user to interact with some example implementations of the current subject matter including showing how the most recommended next step can be illustrated in a predetermined location (e.g., upper right hand neighboring hexagon), with next recommended steps in order of priority proceeding clockwise;

FIG. 16-19 illustrate exemplary data exploration views;

FIG. 20A-20B illustrates an example interface where a user has selected a node and “zoomed” into the node to explore more information about the analytical activity that the node represents;

FIG. 21-26 illustrates a zoom into a data transformation and preparation view, a predictive model development view, and a data exploration view;

FIG. 27-31B illustrate example frames from a movie generated according to some aspects of the current subject matter whereby a user can view the steps that were performed in a given project or analytical task to enable improved traceability;

FIGS. 32A-34B, and 36A-39B illustrate an example audit replay video showing actions taken by users that can be reviewed;

FIGS. 35A-35C and 40A-42C illustrate another example traceability feature according to some example implementations of the current subject matter;

FIG. 43A-44C illustrate example interfaces for a manager role;

FIG. 45A-56B illustrate example interfaces showing an example exchange in which a user can post for micro-task jobs and receive expert help;

FIG. 57A-58B illustrates example interfaces for enabling the system to arbitrate when there is a dispute on how much work the expert performed;

FIG. 59A-62B illustrate some implementations of the current subject matter that can provide an executive overview of various AI and machine learning projects in a given enterprise showing how much value they are creating, how they are linked together, projects that could be creating value but haven't been deployed, and valuable insights from other analytical activities;

FIG. 63 is a flowchart of an exemplary method of displaying a visual representation of input variables of an impact function;

FIG. 64 is an example of a chord chart according to some example implementations;

FIG. 65-67 illustrate examples of the chord chart visualized on a node of several example interfaces;

FIG. 68A-69C illustrate example playback views with differing level of detail in the playback; and

FIG. 70 is a system block diagram illustrating an example implementation of the current subject matter.

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

Analytics (e.g., numerical models, impact functions, etc.) can be used to determine (e.g., predict) an outcome of an analytical process. For example, analytics can be used to model a business process and determine an output that impacts the net profit resulting from the business process. The output of the analytical process can be based on multiple factors (or variables) that can be provided as input variables to a model associated with the analytical process (e.g., a business process). Based on the input variables, the analytical process can generate the output (e.g., as an output of the model) that can change based on changes in the input variables.

The output of a model or analytical process (e.g., a prediction, decision, classification, and the like) can impact a business outcome such as to increase or decrease net revenue, profitability, costs, and the like. But the impact of an output of a model on the business outcome can be complex and can vary based on the value of the outcome. For example, the impact on business outcome of incorrectly predicting that a sales lead should be pursued (e.g., a false positive) can result in an opportunity cost as well as an increased cost of pursing that lead, whereas the impact on business outcome of incorrectly predicting that a sales lead should not be pursued (e.g., a false negative) may result only in lost revenue. In order to more accurately reflect the impact of a model output on a business outcome, an impact function can be utilized to determine the impact a model output will have on the business outcome. An impact function can take as input one or more variables, one or more parameters, and the output of a model or analytical process and determine the impact of the output on the business outcome. For example, an impact function can take as input a number of variable values associated with a sales opportunity lead, parameters associated with the sales team, and the output (e.g., prediction) of a model or analytical process, and determine the impact (sometimes referred to as an impact score) that the output of the model or analytical process will have on the business outcome (e.g., net revenue, profit, costs, and the like). Accordingly, impact functions can account for complex cost-benefit tradeoffs often found in business or other decision-making processes and which may not be taken into account by a model and/or analytical process.

Existing visual representations of analytical processes lack an interface for allowing a user to quickly gain an understanding of the impact of model input variable(s) on the business outcome (e.g., existing visual representations do not provide information for a cost-benefit analysis). Instead, the existing techniques visually represent correlations between input variable(s) and the output of the model. However, identifying and presenting correlation between input variable(s) and output may not be provide adequate information related to, for example, an increase in net profit based on changes in input variables (or a cost-benefit analysis). Similarly, receiver operating curves (ROC) can show the probability of detection (probability of a true positive) and probability of false alarm (probability of a false positive) but do not convey the impact or cost-benefit that a decision will have on business outcome.

It can be desirable to generate a glanceable interface that provides information associated with the input(s) and output(s) of an analytical process in a manner that can allow the user to quickly gain an understanding of changes (e.g., increase or decrease) in the value of the output of an analytical process based on changes in the input of the analytical process. For example, a model can be configured to predict whether to pursue a sales lead based multiple input variables (e.g., number of employees, month of contact with the customers, last contact duration, customer, and the like). It can be desirable to determine the impact on change in net profit to the business using an impact function and present the impact on the change in the net profit based on a pair of variables (e.g., number of employees and last contact duration). This can be done, for example, by weighing the input variable based on their impact on the net profit of the business process.

Accordingly, some implementations of the current subject matter include an interface that enables improved glanceability by providing a visual representation of input variables of an analytical process (e.g., represented by a model) and the effect of the input variables on the output of the impact function. In some implementations, the visual representation is a chord diagram where a portion of the circumference of a circular layout represents each input variable and the chord between a pair of input variables is indicative of the impact of the pair of input variables on the business outcome. For example, a thicker chord can represent a larger impact of the pair of input variables on the business outcome (impact score).

By representing or weighing the thickness of the chords by the relative impact on the business outcome (e.g., the relative thickness of the chord being indicative of the weight those pairs of variables have on the output of the impact function), some implementations of the current subject matter can provide an interface allowing a user to quickly and intuitively understand the relative contribution of a variable to a business outcome.

FIG. 63 is a flowchart of an exemplary method of displaying a visual representation of input variables of an impact function (e.g., representative of an analytical process) and the weight of their impact on the output of the impact function that can, in some implementations, provide for an intuitive interface for understanding the contribution of variables to a business outcome. At step 6302, data characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs is received. The impact function can be configured to receive the input variables and the weights associated with the input variables as input and generate an impact score as output.

In some implementations, the impact function can include a function that enables determining the effect or impact of a decision (e.g., made by a predictive model) on a business organization. For example, an impact function and associated impact score can be used to assess the effect of a decision (e.g., model output, prediction, classification, and the like) on business growth, profit, revenue, resource utilization, and the like. Some example impact functions can characterize maximizing profits for a business, maximizing growth for the business, maximizing revenue for the business, and/or minimizing resource consumption for the business. For example, an impact function can be considered as the sum of the count or probability of true positive, true negative, false positive, and false negative occurrences multiplied by their respective value (or cost where the value is negative) to a business outcome (e.g., revenue, costs).

In some implementations, the impact function can be associated with a model (e.g., representative of an analytical process or task) trained to perform predictions based on the input variables (including input attributes) associated with an entity. The model can be trained, assessed and deployed to provide improved (e.g., optimized, accurate, and the like) prediction for the input variables for achieving a target action (e.g., performing a target function, achieving a target value, and the like). The impact function can calculate an impact score for input variables and the target action. The impact score can be indicative of a prediction for achieving the target action for the input variables. Based on the impact scores, the target action can be assigned or not assigned to the entity with which the input variables are associated.

In some implementations, data characterizing the model and the target action associated with the model can be received. The model can calculate weights associated with one or more input variables (e.g., input variables that are received at step 6302). The weights, the input variables and the target action can be provided to the impact function as an input. Based on these inputs, the impact function can calculate the impact score.

At step 6304, a first coefficient (e.g., interaction coefficient) characterizing interaction between a first input variable and a second input variable of the plurality of input variables can be calculated. The calculation of the first interaction coefficient can be based on a first set of weights of the plurality of weights received at step 6302. The first set of weights can be indicative of interaction between the first input and the second input. For example, the impact function (f(X, Y, Z)) that depends on input variables X, Y and Z can be described as a sum of a plurality of terms:

f(X,Y,Z)=a ₁ XY+a ₂ XZ+a ₃ YZ+a ₄ XY ² +a ₅ X ² Y+a ₆ XZ ² +a ₁ X ² Z

where a₁, a₂, a₃, a₄, a₅, a₆, and a₇, are the weights associates with the impact function. Each term of the plurality term is a product of a weight (e.g., a₁, a₂, a₃, a₄, a₅, a₆, a₇, and the like) of the plurality of weights and at least one input variable (e.g., X, Y, Z, and the like). The first interaction coefficient between input variables X and Y can be based on the first set of weights associated with terms in the impact function that include both X and Y (e.g., the first interaction coefficient can be based on a₁, a₄, and a₅). For example, calculating the first interaction coefficient is based on a first set of terms of the plurality of terms (e.g., a₁XY, a₄XY², and a₅X²Y), where each term of the first set of terms is a product of the first input variable (e.g., X), the second input variable (e.g., Y) and a weight of the first set of weights (e.g., a₁, a₄, a₅, etc.). In some implementations, the first interaction coefficient is the sum of the first set of terms (e.g., a₁XY+a₄XY²+a₅X²Y).

At step 6304, a visual representation (e.g., including one or more graphical objects) of the input variables of the impact function can be displayed in a graphical user interface (GUI). The visual representation can intuitively and quickly provide for a user to analyze the importance of variables and interactions of variables with respect to their relative contribution to a business outcome. A graphical object in the visual representation can include a plurality of regions where each region is representative of an input variable (e.g., a first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively).

In some implementations, the graphical object can include a chord chart. FIG. 64 illustrates an exemplary chord chart 6400. The chord chart 6400 includes a circle where portions of the circumference (e.g., regions shown as arcs 6402-6426) can represent input variables. The relative size of the arcs represent the importance of the input variables (e.g., large being more important). For example, arc 6406 (representative of input variable “Last Contact Duration”) is longer than the arc 6408 (representative of input variable “Age”). This indicates that input variable “Last Contact Duration” 6406 has a greater impact on the output of the impact function (e.g., impact score) than input variable “Age” 6408.

Returning to FIG. 63 , at step 6308, a graphical object (e.g., a second graphical object) connecting the first arc to the second arc can be displayed in the GUI. The second graphical object can be a chord connecting a portion of the circumference (e.g., first arc) to a second portion of the circumference (e.g., second arc). The chords connecting the variables can show the interaction (e.g., cost-benefit tradeoff, impact, and the like) between the input variables with respect to their relative contribution to a business outcome. The chord can be weighted by economic impact of another factor, and not just statistical correlation. The visual characteristics of the chord connecting the first arc (representative of the first input variable) and the second arc (representative of the second input variable) can be representative of impact (e.g., strength) of the combination of first and second input variables on the output of the impact function (e.g., impact score). For example, the visual characteristic (e.g., thickness of the chord) can be related (e.g., proportional) to the value of the interaction coefficient associated with the first and the second input variables. In some implementations, the length of a chord that connects back to itself shows the strength or contribution of the individual variable on the impact score. The chords can give a quick overview of which single and multi-variable combinations are most important drivers of the business outcome (e.g., impact score) such as win rate or infection rate.

These drivers can be measured based on simple statistical measures like prediction drivers/regression coefficients and the like. In some implementations, the chords can be weighted by expected business impact by weighing the chords and arcs by (a) relative occurrence of success/failure states, (b) expected business impact of the specific driver. Business impact of the specific driver is related to quantifying how much impact would be delivered by an artificial intelligence (AI) (e.g., a model or analytical process) or how important the driver is to the business definition of success. For example, in a sales use case, a successful sale may be worth $100 and a failed attempt at a sale may cost $1. The statistical driver as calculated by standard techniques for detecting the importance of a prediction factor such as ‘Industry=Manufacturing” may thus be multiplied by a) the relative frequency-weighted relative impact ($100 vs. −$1) of the occurrence of success and failure states in the data where Industry=Manufacturing or b) the relative frequency-weighted relative impact ($100 vs. −$1) of the expected True Positive and False Positive rates for a predictive model if applied to predict sales opportunities where Industry=Manufacturing in the future. This allocates that expected impact back to the different drivers of the AI. So if one variable is responsible for 10% of the relative driver strength of the AI, then it would be weighted at 10% in the chord diagram.

Although a few variations have been described in detail above, other modifications or additions are possible. For example, although the term “business outcome” has been used throughout, in some implementations, the current subject matter is not limited to business and can be applied to a wide range of fields and problems such as public health (e.g., determining impact on infection rates), medicine (e.g., determine impact on individual health), public policy (including environmental benefit and resource efficiency of specifically environmental protection actions), disaster relief (including equitable allocation of scarce resources such as food, water, relief supplies, and the like), and any problem that modeling and analytical processing can be applied.

Although the example implementation illustrated a chard diagram, the current subject matter can take other forms such as two parallel lines with portions (e.g., line segments) representing variables and lines connecting the portions with thicknesses representing relative impact of pairs of variables (such as in a Sankey chart where the thickness of the lines connecting the two portions are weighted by or representative of relative impact). The current subject matter can be implemented in a Sankey Chart, a heat map, a bubble chart, a network graph, and the like. In some implementations, current subject matter can be implemented as a heat map, which is a data visualization that shows values as color in two dimensions. The location of a graphical object (e.g., box) can be indicative of two variables and the color of the graphical object may be representative of the relative impact between the two variables. In some implementations, the current subject matter can be implemented as a bubble chart, where the location of a bubble is indicative of two variables and the size of a bubble is representative of the relative impact between the two variables. In some implementations, the current subject matter can be implemented as a network graph, which is a visualization that shows relationships between entities in graph form. The nodes of the graphs can be representative of variables and the thickness and/or color of lines connecting nodes can be indicative of relative impact of the two variables corresponding to the nodes being connected by the line. Other forms are possible.

In some implementations, the thickness of the chord can be indicative of contribution of two variables to one or more of frequency (of an outcome); counts; costs or benefit of a specific error or success; factors related to the cost or benefit of the specific error or success; frequency, counts, or costs of prediction errors; and the like. For example, the chord diagram can visually show where the most false positives are happening (e.g., which two variables), where false positive cost is highest, or where the false positive frequency is highest. Similar visualizations can be made for true positives, false negatives, and true negatives. As another example, the chord diagram can visually show the frequency, count, or costs of a recommendation (made by the system and/or a model) being followed or not followed by an end user. In some implementations, the portions of the graphical object related to variables can be unrelated to modeling and instead relate to other variables, such as population counts and concentrations. Other implementations are possible.

The following describes an example analytical task platform for developing analytical tasks that can utilize an example chord diagram as described above.

Analytical Platform

Analytical tasks (such as model building, assessment, and deployment) can include performing complex analytical processes using systems such as model building platforms that enable the creation of analytics (e.g., models). These techniques and systems may record a log of actions taken by a user (e.g., a model developer) but the log can be difficult for the user to follow, particularly where the user is a business user, rather than a technical subject matter expert. Moreover, the user may not be able to understand what steps they or another user has taken in developing the analytics. Analytical tasks can be composed into granular analytical tasks that can be carried out independently or dependently in the pursuit of the overall analytical goal. Decomposing the overall analytical task into granular tasks can simplify context transfer and collaboration. Thus some existing analytical systems lack an interface for allowing a user to quickly gain an understanding of a given analytic task (e.g., model, process, and the like), quickly gain an understanding of what may have changed in an analytic over a period of time, and quickly gain an understanding of how an analytic was developed. The ability of a system for a user to quickly gain an understanding of analytics, changes to the analytics, and how the analytics were developed can be sometimes referred to as “glanceability” of the model building and deployment system. More broadly, whether an interface is glanceable can refer to information on an electronic interface that can be understood quickly or at a glance.

Some implementations of the current subject matter include an interface that enables improved glanceability, including by providing an intuitive interface into a model building and deployment system. The interface can include a graphical user interface that can include nodes (for example hex boxes as shown in the examples below) that can represent granular analytical tasks and convey information associated with those tasks, such as the impact on the project or enterprise from the granular analytical task. The nodes can be arranged in a manner to convey relationships among the analytical tasks. For example, one project may be comprised of multiple analytical tasks, which themselves may have sub-tasks. Each of those sub-tasks may have multiple associated actions (e.g., sub-tasks of the sub-task), and so on. By displaying analytical tasks graphically using nodes, where the content, characteristics (e.g., color, shading, and the like), and arrangement of the nodes can allow for conveying complex information about the analytical tasks quickly and intuitively.

The nodes can be interactive, allowing for selection of the node to, for example, control actions in the performance of the analytical task, such as allowing for exploration of information associated with the node (e.g., viewing graphs and other analysis related to the selected analytic task, including display of an impact chord diagram such as is described above with respect to FIGS. 63 and 64 ), providing an interface for performing any incomplete analytic tasks, and the like.

Views

In some implementations, a user can interact with a graphical user interface that can be representative of (and allow for the performance of) a plurality of analytical tasks (e.g., a plurality of analytical tasks associated with a multi-step analytical process). For example, a node in the GUI can be indicative of a first analytical task that has been performed. In order for a user to perform a second step in the multi-step analytical process, the user can select a second analytical task from a first set of possible or available analytical tasks, which can be subsequently performed via the interface.

A user can request adding a node representative of the second analytical task in the GUI, for example, by selecting an existing node in the interface. In some implementations, when the user requests the addition of the second analytical task (e.g., by selecting an existing node), the GUI can generate a first set of nodes adjacent to the first node (e.g., where each node of the first set of nodes represents an analytical task from the first set of analytical tasks that can be performed within the system). The user can select the desired second analytical task to be performed from the first set of analytical tasks by selecting the corresponding node (e.g., second node) from the first set of nodes in the GUI. In some implementations, after the selection of the second node, the remaining nodes from the first set of nodes is no longer be visible, indicating that the second node has been selected by the user. A subsequent node (e.g., a third node, a fourth node, etc.) can be added by repeating the above-mentioned method.

The arrangement of the first set of nodes in the GUI can be indicative of a priority associated with the nodes. In some implementations, a priority of the node can be representative of a recommendation by the system for a next step to be performed in the multi-step analytical process. In some implementations, the first set of nodes can be arranged adjacent the first node based on the priority of the analytical tasks corresponding to the nodes in the first set of nodes. For example, a node arranged above and to the right of the primary node can represent a highest priority step, and nodes arranged downstream in the clockwise direction can have decreasing (or lower) priority.

FIG. 1 illustrates an exemplary process of generating an interface that can enable performance of steps in a multi-step analytical process, viewing of previously-completed steps in the multi-step analytical process, and other functionality in an intuitive manner for a user. At step 102, a first node having a first location can be provided in a first graphical user interface (GUI) display space. FIG. 2 illustrates an exemplary GUI 200 that includes a first node 202. The first node can be indicative of a first analytical task. In some implementations, the first analytical task can serve as a first step of a multi-step analytical process. For example, the multi-step analytical process can include importing a dataset, building a model using the dataset, and/or deploying the model to operate on live data. Other actions can be performed in the multi-step analytics process, as described more fully below. Subsequent steps of the multi-step analytical process can be performed utilizing the GUI 200, which can include displaying nodes to represent the subsequent steps of the multi-step analytical process.

At step 104, data characterizing a first user input indicative of selection of the first node of the first set of nodes can be received. For example, the first node 202 can be interactive and the user can interact with it (e.g., by clicking on it). Based on the user interaction with the first node by the user, the first analytical task represented by the first node 202 can be performed. In some implementations, performance of the fist analytical task can be performed via another screen or view of the GUI 200.

In some implementations, the user can interact with the first node to indicate adding of a next step of the multi-step analytical process. The next step can include performing a second analytical task that can be performed after the first analytical task in the multi-step analytical process. In some implementations, the next step can include importing a dataset, building an analytical model (e.g., included in the analytical task) using the imported dataset, deploying the analytical model to operate on live data, and the like.

At step 106, the first set of nodes can be displayed in the first GUI display space. The first set of nodes can represent steps in the multi-step analytical process that are available to be performed. FIG. 3 illustrates the first GUI display space 200 that includes the first set of nodes 204-218 that are displayed adjacent to the first node 202. Each node of the first set of nodes 204-212 is associated with a possible next step of the multi-step analytical process (e.g., node 204 can represent a marketing task, node 206 can represent a sales task, node 208 can represent a churn task, node 210 can represent a customer service task, node 212 represents a supply chain task, etc.).

The arrangement of the nodes in the first set of nodes in the first GUI can be representative of the priority of the analytical task represented by the corresponding node. In some implementations, the nodes of the first set of nodes can be arranged clockwise (relative to the first location of the first node) in the decreasing order of priority. For example, node 204 located at a second location in the interface 200 can have the highest priority, node 206 located at a third location (adjacent to the node 204 in a clockwise direction (around the first location of the first node) from the second location of the node 204) and can have the second highest priority, and node 212 located at a fourth location in the interface 200 can have the lowest priority. In some implementations, the nodes of the first set of nodes can be arranged counter-clockwise (relative to the first location of the first node) in a decreasing order of priority. For example, node 212 can have the highest priority, node 210 located at a fifth location (adjacent to the node 212 in an anti-clockwise direction (around the first location of the first node) from the third location of the node 212) and can have the second highest priority, and node 204 can have the lowest priority. In some implementations, priority can be determined by the system, for example, utilizing predictive models that predict the next-best step to be performed based on historical user activity for users with similar objectives or performing similar multi-step analytical processes. In some implementations, the priority order can be predefined or predetermined. In some implementations, there is no priority order to the displayed nodes representing possible next-steps in the multi-step analytical process.

In some implementations, visual characteristics of a node can be indicative of various properties of the analytical task associated with the node. In some implementations, the boundary of a node can indicate whether the analytical task has been completed/selected or not. For example, if the boundary of the node (e.g., node 204) is a solid line, it is indicated that the corresponding analytical task (e.g., analytical task indicated by the node 204) has been completed/selected. As illustrated in FIG. 3 , the boundaries of the first set of nodes 204-212 are represented by a dashed line. This can indicate that no node from the first set of nodes 204-212 has been selected yet. In other words, the analytical tasks associated with the first set of nodes are the possible options for the next analytical task (after the first analytical task associated with the first node 202), and the user has not made a selection for the next task.

FIG. 4 represents the exemplary GUI 200, where the user has selected the analytical task (associated with the node 206) as the second task (or the next task) of the multi-step analytical process. Upon selection, the boundary of the node 206 is represented by a solid line. The user may decide to continue building the multi-step analytical process. This can be done, for example, by adding a third analytical model to the existing first analytical model (represented by first node 202) and the second analytical mode (represented by second node 206). The user can interact with the second node 206 (e.g., by clicking on it) to indicate, for example, the generation of a next step (or the third step) of the multi-step analytical process. The next step can include generating the third analytical task that can be executed after the execution of the first and the second analytical task in the multi-step analytical process.

When the user interacts with the second node 206, data characterizing the user's interaction with the second node 206 is received. Based on the receipt of user's interaction, a third set of nodes 220-224 can be displayed adjacent to the second node 206. Each node of the third set of nodes 220-224 are associated with a possible next step of the multi-step analytical process. As described above, in some implementations, the arrangement of the nodes in the third set of nodes in the first GUI 200 can be representative of the priority of the analytical task represented by the corresponding node. For example, node 220 can have the highest priority, node 222 can have the second highest priority and node 224 can have the third highest priority. Alternately, node 224 can have the highest priority, node 222 can have the second highest priority and node 220 can have the third highest priority. The boundaries of the third set of nodes 220-224 are represented by a dashed line. This can indicate that no node from the third set of nodes 220-224 has been selected yet.

FIG. 5 represents the exemplary GUI 200, where the user has selected the analytical task (associated with the node 220) as the third task (or the next task) of the multi-step analytical process. Upon selection, the boundary of the node 220 is represented by a solid line. The user may decide to continue building the multi-step analytical process. This can be done, for example, by adding a fourth analytical model to the existing first analytical model (represented by first node 202), the second analytical mode (represented by second node 206) and the third analytical model (represented by the third node 220). The user can interact with the third node 220 (e.g., by clicking on it) to indicate, for example, the generation of a next step (or the third step) of the multi-step analytical process. The next step can include generating a fourth analytical task that can be executed after the first, the second and the third analytical task in the multi-step analytical process.

When the user interacts with the second node 220, data characterizing the user's interaction with the second node 220 is received. Based on the receipt of user's interaction, a fourth set of nodes 230-234 can be displayed adjacent to the third node 220. Each node of the fourth set of nodes 230-234 is associated with a possible next step of the multi-step analytical process. As described above, in some implementations, the arrangement of the nodes in the fourth set of nodes in the first GUI 200 can be representative of the priority of the analytical task represented by the corresponding node. For example, node 230 can have the highest priority, node 232 can have the second highest priority and node 234 can have the third highest priority. Alternately, node 234 can have the highest priority, node 232 can have the second highest priority and node 230 can have the third highest priority. The boundaries of the fourth set of nodes 230-234 are represented by a dashed line. This can indicate that no node from the fourth set of nodes 230-234 has been selected yet.

FIG. 6 is an image of an example graphical user interface 600 according to some example implementations of the current subject matter that can provide an intuitive interface to managing and performing analytical tasks using a model building and deployment platform. The illustrated example interface 600 shows a summary or overview view. The initial node 602 (dark purple node) is illustrated with three connectors icons that connect the initial node (dark purple) to three neighboring nodes (labeled marketing 604 (light purple), labeled Sales 606 (light blue), and labeled Churn 608 light green). The initial node 602 (dark purple node) includes text describing a metric, in this case, the value that the analytical system is providing to an enterprise. In some implementations, the summary view can summarize one or more projects, which can include a collection of analytical tasks devoted to accomplishing a certain task, limited to a certain business unit, operation, team, and the like. As illustrated in FIG. 6 , each node or hex box can represent a project, which can interact with one another.

The three neighboring nodes are projects including analytical tasks that have been completed and deployed by a user implementing the analytical tasks. The border of these nodes is solid indicating that the analytical tasks are completed and deployed. Nodes with dashed borders can indicate that an analytical task was completed but not deployed by the user, or that the analytical task or project was started but is still incomplete.

Each completed neighboring node (Marketing, Sales, and Churn), also shows that tasks' impact on the overall project or enterprise (e.g., on the initial node). For example, the Marketing, Sales, and Churn nodes also indicate the monetary value of those projects and associated analytical tasks to the enterprise.

Thus, the example interface 600 can provide a significant amount of information to a user relating to the analytical tasks and projects at a “glance” of the summary view.

FIG. 7 is another example of the summary view interface as described with respect to FIG. 6 .

FIGS. 8-10 illustrate an example interface for building a predictive model using the example expandable node approach, for example, a model building view. With reference to FIG. 8 , a user can begin with selecting a blueprint, selecting data sources, and then training an AI. Additional actions such as predicting baseline behavior (e.g., model performance assessment), strategy, integration, and impact monitoring can also be implemented by the user.

In some implementations, a user can work on the analytical task associated with a node. For example, the user can select one of the nodes 204-212 in interface 200 and add analytical sub-tasks to the analytical task associated with the selected node. In some implementations, a second user input indicative of an interaction (e.g., selection) with a node (e.g., node 206 associated with the sales task) can be received. Based on the interaction a second representation graphical user interface (e.g., interface 800) can be displayed which can allow for building the analytical task (e.g., a predictive model in the analytical task) associated with the interacted node. For example, the interface 800 can allow for building on the predictive model of the sales task. As illustrated in FIG. 8 , interface 800 includes nodes 802-804 that represent a plurality of analytical sub-tasks associated with the sales task. Building the sales task can include retrieving/adding sales data (represented by node 802), adding a sales blueprint (represented by node 804), performing AI training (represented by node 806), predicting baseline behavior (represented by node 808), applying a strategy (represented by node 810), integrating the sales task with an existing artificial intelligence model (represented by node 812), performing impact monitoring (represented by node 814), etc. The nodes are placed based on the temporal order in which they were created (e.g., node 802 is created first and node 814 is created last). In some implementations, a new node representing an analytical subtask can be added by a user. For example, a user input indicative of addition of a new analytical sub-task to the set of analytical sub-tasks can be received (e.g., via interface 800). The new node (e.g., node 814) is placed adjacent to a previous node (e.g., node 812 that is the last node to be generated temporally prior to the generation of node 814) in the interface 800. The previous node can be indicative of a previous analytical sub-task (e.g., artificial intelligence model associated with node 812) of the plurality of analytical sub-tasks. The location of a new node relative to the previous node can be indicative of the priority of the analytical sub-task associated with the new node. For example, as the priority of the analytical sub-task decreases, the location of the new node relative to the previous node is rotated in the clockwise (or anti-clockwise direction).

FIG. 11 is another example interface 1100 illustrating data transformation and data preparation views. The illustrated nodes can represent data sources and functions on those data sources. Nodes that have common borders can show that a function (e.g., nodes labeled f(x), where a function can be any function that would be applied to a data source, such a “x²”) has been applied to the data source. A node illustrating a circle with a line through it can show that four variables have been excluded from the associated data source. The icon (plus sign) between the two data sources can indicate that the two data sources (with their associated functions applied and variables excluded) have been joined. The shaded nodes can represent recommendations or options for manipulating the data source or applying an analytical task, and selection of these nodes by the user can implement the associated analytical task. FIG. 12 is another example interface as described with respect to FIG. 11 . FIG. 13A-13C illustrates an example of data transformation. FIG. 14 illustrates an interface for selecting different projects or creating a new project.

In some implementations, interface 1100 can be displayed based on a user input indicative of selection of a node in interface 200 or interface 800. For example, the interface 1100 is displayed based on interaction (e.g., selection) with a node (e.g., node 802 associated with the analytical task of adding sales data) from the plurality of nodes (e.g., nodes 802-814) in the interface 800. The interface 1100 can include a subset of node that includes node 1102 indicative of a dataset (e.g., sales dataset associated with the analytical sub-task of node 802) and node 1104. Node 1104 can be located adjacent to the node 1102 (e.g., share a common border) and can be indicative of a function or an operation that can be (or has been) applied on the dataset. For example, a common border between node 1102 and the node 1104 can be absent. This can indicate that the operation associated with node 1104 has been applied on the dataset associated with node 1102.

In some implementations, a third node can be added to the subset of nodes. The third node can provide additional information associated with the application of the operation on the dataset (e.g., portion of the dataset that has been excluded from the application of the operation). For example, three nodes 1106, 1108 and 1110 can form a subset of nodes where one or more borders missing between the nodes in the subset. For example, a first border between node 1106 and 1108, and a second border between node 1108 and 1110 can be missing. The subset of nodes can indicate that an operation (associated with node 1106) has been applied on a portion of the dataset (associated with node 1108). The numerical value in the node 1110 can indicate the portion of the dataset that has not been operated upon by the operation associated with node 1106.

In some implementations, a graphical user interface (e.g., graphical user interface 200) can represent a multi-step analytical process that can include a hierarchy of tasks. The hierarchy can include multiple layers where each layer can include multiple analytical tasks. For example, a first layer can include an analytical task and a second layer can include a second layer analytical sub-tasks associated with the analytical task in the first layer (e.g., the second layer analytical sub-task can sequentially follow the analytical task in the first layer). In some implementations, the hierarchy can include a third layer that can include third a layer analytical sub-tasks associated with at least one analytical task in the second layer.

In some implementations of the current subject matter, the hierarchical multi-step analytical process can be visually represented. In some implementations, the first layer of the hierarchy can be represented by a first representation of a GUI, the second layer of the hierarchy can be represented by a second representation of the GUI, and a third layer of the hierarchy can be represented by a third representation of the GUI. For example, the first layer of the hierarchy can be represented by the GUI 200, the second layer can be represented by GUI 800, and the third layer can be represented by the GUI 1100. As illustrated in GUI 200, an analytical tasks in the first layer (analytical task associated marketing, sales, churn, customer service, supply chain, etc.) can be represented by one of the nodes 204-212.

In some implementations, the user can access the analytical tasks in the second layer by interacting with one of the nodes 204-212. For example, the user can interact with node 206 associated with sales, and the GUI 800 can be presented that includes a visual representation (e.g., one or more nodes) of the second layer analytical sub-tasks associated with the sales analytical task (represented by node 206). The second layer analytical sub-tasks can include, for example, importing data, training models, assessing, deploying, monitoring the deployment, etc. For example, a second layer analytical sub-tasks can be represented by one of the nodes 802-814.

In some implementations, the user can access the analytical tasks in the third layer by interacting by one of the nodes representative of second layer analytical sub-tasks. For example, the user can interact with node 802 associated with addition/importing of sales data, and the GUI 1100 can be presented that includes a visual representations (e.g., one or more nodes) of third layer analytical sub-task associated with addition/importing of sales data (represented by node 802). As described later, GUI 1100 can allow the user to identify the data to import and perform basic functions/joins/etc. on the data.

The interaction with a node that allows the user to move from one layer to another (e.g., from the first layer to the second layer) can be different from the user interaction associated with addition of a next step of the multi-step analytical process (e.g., as described in step 104 above). For example, in order to switch views, a user may double click a node, select an action from a context menu, select an action from a navigation bar, and the like.

In some implementations, the current subject matter can enable conveying recommendations visually, for example, by arranging nodes around a selected node in a predetermined order. For example, in some implementations, the model development and deployment system can provide recommendations to the user regarding which analytical tasks would be most impactful on the performance of the project or enterprise. These recommendations can be provided by displaying new nodes surrounding a current node, where each new node corresponds to a recommended action. The arrangement of new nodes can indicate the relative order of the recommendations. For example, recommendations can be arranged in a clockwise order.

FIG. 15A-15B illustrates a recommendation interface 1500 that can provide recommendation (or information) for building an interface (e.g., interface 200) of a multi-step analytical process. In some implementations, an input from a user indicative of a request for recommendation (or information) can be received (e.g., via interface 200). This can be done prior to or during the generation of the nodes in the interface 200. After receiving the request from the user, the recommendation interface 1500 can be generated. The recommendation interface 1500 can include multiple recommendation nodes 1502-1510 that can include recommendation/information for building the interface representative of a multi-step analytical process. For example, a first recommendation node 1502 can be indicative of a starting node (e.g., node 202). The starting node can be associated with an analytical task in the first layer of the multi-step analytical process hierarchy. The recommendation interface 1500 can include a first set of recommendation nodes 1504-1510 that can be indicative of nodes (e.g., nodes 204-210) associated with analytical tasks in the second layer of the multi-step analytical process hierarchy. The recommendation nodes in the first set of recommendation nodes can includes properties of analytical tasks in the second layer. The properties can include, for example, description of an analytical task (e.g., as text inside the corresponding recommendation node), priority level of the analytical task (e.g., represented by location of the corresponding recommendation node relative to the first recommendation node 1502), etc.

In some implementations, a user can add graphs to a data exploration analytical task, which can be illustrated, for example by a number (FIG. 16 ) or by showing stacked nodes (FIG. 17 ). Nodes that are adjacent can correspond to related subjects whereas non-adjacent nodes can correspond to a different subject. Additional data exploration views are illustrated at FIGS. 18 and 19 .

As noted above, the direction that nodes are presented as a user is performing the analytical task can relate to actions the system recommends and they can be presented in an order of priority of recommendation. For example, the most recommended next action can always be provided at the upper right edge of a node (e.g., the “1 o'clock” position), and subsequent recommendations can be displayed in clockwise direction. Thus, a user reviewing an already performed analytical task can quickly infer from the structure of the displayed node graph whether the recommendations were followed. Thus if a node graph shows nodes extending generally in the upper-right direction from the initial node (for example as shown in FIG. 8 ), then it can be inferred that the user generally performed the most recommended actions. Similarly, node graphs that show nodes extending generally in the left direction from the initial node can indicate that the user did not follow the actions recommended by the system.

Further, the order in which the user performed the steps of the analytical task, or collection of analytical tasks, can be reflected by the shape of the node graph. For example, starting at an initial node, the next performed action can be represented by an adjacent node. Thus the node graph visualization can provide a quick and intuitive display for understanding not just what actions a user has performed but in what order.

These different above-described views can communicate several types of additional valuable information. For example, colors in data exploration view can show which variables the user or automated analysis focused on (e.g., each color or numbered hexagon showing how many charts focused on each variable were added to the overall analysis); grouping or insights be conveyed, for example, in FIG. 19 , a core analysis was conducted spanning various variables and then a separate set of four graphs focused on two variables; whether a chart added to the exploration was recommended by the system or added manually by the user; whether the user follow system recommendations or not; and the like.

Zooming within a View for Additional Details

FIG. 20A-20B illustrates an example interface where a user has selected a node and “zoomed” into the node to explore more information about the analytical activity that the node represents. As illustrated, a graph or other display space can be rendered enabling consumption of information in a variety of modes. The zoomed in view can take a number of forms, which can vary based on the view type. For example, FIG. 20A-20B illustrates a zoom into a summary view, FIGS. 21-22 illustrates a zoom into a data transformation and preparation view, FIGS. 23-24 illustrate a zoom into a more detailed view of a predictive model development view, and FIG. 25-26 illustrate a zoom into a more detailed view of a data exploration view.

Collaboration Between Users and Tracking Changes to Analytical Tasks

Some implementations of the current subject matter can enable multiple users to collaborate on the same analytical tasks. Both the glanceable views and zoomed views can show how and when multiple users collaborated. Users can also easily hand off an analytical task to an expert and then take it back without losing the context of what the expert did. For example, in the midst of an exploration a business user may not know where to go next. They ask an expert to help. The expert looks at different parts of the analysis already conducted and then starts adding a few different charts that can be good starting points for additional exploration. They then hand the project back to the original user who can see exactly what the expert user has done via a ‘movie-like playback’ and can see the key charts the expert user tagged as good starting points for further collaboration. Either user can comment on any chart or any aspects of the analytical task.

FIG. 27-31 illustrate example frames from a movie generated according to some aspects of the current subject matter whereby a user can view the steps that were performed in a given project or analytical task to enable improved traceability. By enabling a “replay” of actions performed, collaboration between users can be improved. Moreover, such a reply can enable improved contextual understanding of a given analytical task, for example, if a user has not worked on a particular analytical task in some time. In some implementations, different lines at the bottom of the interface in FIG. 27-31 can relate to actions taken by different users.

In some implementations, the model development and deployment platform can regularly provide recommendations to the user regarding what actions to take next or how to complete a given action. Many recommendation provision systems (e.g., recommendation engines) are driven off of a knowledge graph based on data. In some implementations, the system can also utilize a human interaction graph. The human interaction graph can be a knowledge representation based on behavior of users with the system. For example, the system can learn that a particular graph or analysis is performed within a certain analytical task, and the system can learn the user's behavior and consider that human interaction graph for preparing the recommendations. The human interaction graph can be determined as a variety of granular populations. For example, the human interaction graph can be created from the collective action of all users of the system, all users within a business, all users within a team of the business, or based on the actions of the individual. In this manner, the system can automatically develop domain specific knowledge regarding best practices based on monitoring user behavior and utilizing the user behavior to affect the recommendations that the system provides. In some implementations, the user behavior including historical analytical tasks performed by a user can be saved off as a blueprint for future projects and/or analytical tasks.

Audits and Traceability

Some implementations of the current subject matter can enable user friendly traceability and auditability for analytical activities such as data preparation, data exploration, predictive model building. As noted above, analytical activities can be complex and full audit or traceability logs of such activities can be difficult to navigate. As such, it is often the case that such logs are only used by experts in extraordinary circumstances like evidence in a lawsuit. However, traceability and auditability can provide an answer to the ‘how’ behind the analyses, charts, datasets that users create—how exactly was this dataset created, or how exactly did an analyst arrive at this conclusion—not just what they added to a report, but also what they explored but didn't add to the report. Some implementations of the current subject matter can make auditability and traceability easy for end users, which can generate greater trust in analytical output. Moreover, easy auditability and traceability can enable users to effectively collaborate with each other because they can quickly get a sense of what each person has done and what tasks remain undone. And even when a single user approaches the same analytical task after a period of time, they often forget what they had done to arrive at the output of the analysis task. Being able to quickly review how they got to that point when they had previously worked on the task, helps them quickly regain the context so that they can continue or adjust the original analytical task.

In some implementations, all of the tasks conducted by users are tracked and shown on a timeline. Users can play back the movie of the analytical task and see exactly what was done, what was considered but not acted upon, what recommendations were followed and not followed, as well as the next recommended steps to take in the exploration. Glanceable visualization helps them see what was done at a high level and they can quickly zoom in at any point in the playback to ascertain greater details.

FIGS. 32A-34, and 36-39 illustrate an example audit replay video showing actions taken by users that can be reviewed. FIG. 32A-32B illustrates an overview and FIG. 33A-33B illustrates an example project creation start. Hexes show guidance on where to start, but timeline is empty since no actions have been taken. FIG. 34A-34B illustrates that the first user actions are logged. In the figure, hexes show that data is added, artificial intelligence (AI) is created and deployed. The center hex summarizes the impact of the project. Users can review actions by replaying the timeline at the bottom left. FIG. 36A-36B illustrates the interface after a user has added a blueprint and perform an insight exploration. FIG. 37A-37B illustrates that next a new user does not make any changes but gives a lot of feedback (red or highlighted hexes can indicate that a comment is provided and associated with a given hex). FIG. 38A-38B illustrates that another user updates the data, creates and deploys AI. As shown in FIG. 39A-39B, when the video is replayed, the content shifts to each section, replays edits (e.g., insight exploration in this example), and a list of the users making current updates is shown in the panel on the lower left.

FIGS. 35 and 40-42 illustrate another example traceability feature according to some example implementations of the current subject matter. FIG. 40A-40B illustrates an overview that shows all actions that have been taken in an AI project. FIG. 41A-41C illustrates that the user clicks on the data (green) section and it focuses on those actions. FIG. 42A-42C illustrates the user clicks on the AI creation (purple) section and traceability switches focus to that area. FIG. 35A-35C illustrates what happens when a user clicks into the AI creation hex before clicking into the data prep hex. A user can take action to add a blueprint, explore data, and create two new AI updates. Actions impact is logged on the left.

Management Overview

Some implementations of the current subject matter can provide a management overview of the analytical activities of business users showing productive work done on analytical activities such as data preparation, data exploration, predictive model building while maintaining traceability and auditability and without giving users access to the raw data.

FIG. 43A-44C illustrate example interfaces for a manager role. It shows various employees, how many datasets they have worked on this week, how many explorations, and the like. They can see icons for each employee. They can see before and after icons for things that have changed. They can zoom in and they can play the movie. FIG. 43A-43B illustrates an interface in which a manager sees an overview of all the activities that have been done within their organization. FIG. 44A-44C illustrates an interface in which clicking on an activity opens the timeline replay of what the user did during that session. In another implementation of the interface, managers can see the tasks backlog of each employee such that they can easily review what tasks were assigned to an employee during a time period, what tasks they completed, and what remains to be done.

Enabling Independent Expert Input for Micro-Analytical Tasks

Moreover, as described more fully below, using the intuitive interface for analytical tasks, additional functionality can be achieved to enable a user (such as a non-technical business user) to obtain independent or outside expertise regarding their analytical tasks in a manner that can (1) protect data security of the analytical tasks since the outside expert can review the user's interface (e.g., node graph) without having access to the underlying data; (2) enable quick understanding by the expert regarding which steps the user has and has not performed in an analytical task; and (3) allow for an independent evaluation of work or advise given by the expert.

In some implementations, the user interface allows for auditability and traceability of analytical tasks. For example, using the improved interface, a user can “replay” actions that they took to perform an analytical task (such as building a predictive model). The system can show, by visualizing the node graph expanding as the user performed prior actions, the information contained in the audit log in an intuitive and easily understood (e.g., glanceable) manner. Such an approach can enable the non-technical business user to understand past analytical task actions to gain further understanding of a project. Further, such an approach can enable another user (such as an auditor, manager, expert, or collaborator) to understand past analytical task actions to gain further understanding of a project, thereby enabling them to work on the project (e.g., collaborate, audit, provide advice, perform quality control, and the like). The “replay” can be user-role-specific such that a business user would see a different level of details compared to an expert user even as they view a replay of the exact same set of analytical tasks. Examples interfaces showing differing levels of detail are shown at FIGS. 68 and 69 .

Some implementations of the current subject matter can enable a services exchange where customers can easily arrange for analysts (e.g., subject matter experts) to work on specific analytical activities such as data preparation, data exploration, predictive model building while maintaining traceability, auditability, and without giving analysts access to the raw data. In some implementations, the system can enable enforceable satisfaction guarantees (e.g., an independent evaluation of work performed by expert).

In some implementations, a requirements document (e.g., job description) can be automatically generated based on actions by a user within a project. For example, a user who encounters difficulty in completing an analytical task in a project, for example, trying to add additional fields from a data set, can request a requirements document (e.g., job description) be automatically generated. Because the system understands where within the project and analytical task the user is currently working, the tasks the user has already completed, the tasks they worked on but did not complete, and the recommended tasks to be done, the system can automatically generate the requirements document. For example, the auto generated requirements document can request that “current dataset has 15 fields from CRM and marketing. Looking to add 5-10 additional fields.” In one example implementation, the fact that ‘5-10 additional fields’ may need to be added can be generated by comparing the dataset to other datasets used for successful analyses of the same sort. Such benchmarking can be specific to the user's organization, the use case, the specific user's other datasets, etc. The requirements document can be posted to an exchange on the system for experts to perform the requested micro-tasks.

Once an independent expert has accepted the job request, they can determine the current context of the project using, for example, a replay and collaboration feature as described above that allows another user to view the steps taken in the currently worked upon analytical task. By enabling the expert to quickly and intuitively understand the context of the currently worked upon project or analytical task, it enables experts to provide input for micro-tasks quickly and efficiently.

Some implementations of the current subject matter can also provide for data security. While a user may require expert help for a particular task they may not want to expose their data to an independent expert. By utilizing the interfaces described herein, some aspects of the current subject matter can allow for collaboration between an expert and a user without exposing the underlying data, thereby improving data security, because the analytical task can be performed entirely within the system the underlying raw data is never exposed or made available to the independent expert. The data can be stored securely in a cloud environment specific to the data provider and all analytical tasks of the expert translates into the code that executes in the data-provider's cloud account. Such code can be restricted from transferring anything other than high-level query results and specifically prevented from giving access to raw data. In one implementation, an approach such as k-anonymity can be implemented such that the system will only allow aggregation queries on data subsets where the count is greater than k.

Further, some implementations can allow for tracking of the expert's activities thereby giving the user assurance regarding the amount of time and steps taken by the expert when working on a particular job. Such tracking can be performed, for example, by monitoring interactions of the expert with the systems' servers. Such assurance can enable automated satisfaction guarantees (e.g., an independent evaluation of work performed by expert) and in a manner that does not log the entire view (e.g., user interface) of the expert, thereby avoiding privacy concerns associated with screen loggers.

FIG. 45A-56 illustrate example interfaces showing an example exchange in which a user can post for micro-task jobs and receive expert help. FIG. 50 illustrates an interface that enables, once the user has hired a consultant for a project, to securely share the data via a data portal. FIG. 51A-51C illustrates that if a user has an active project, explorations, and the like linked to their account they will see recommended experts for their ongoing projects. FIG. 52A-52C illustrates that if a user is working on a project they can click on the “ask an expert” hexagon and see the list of available (and recommended best match) experts listed on the left hand side. The recommendation may be based on expert ratings but also based on their expertise on the specific kind of project the user is requesting for help on. Thus, if the user is asking for help on a sales optimization use case, priority would be given to experts who specifically were rated well on such use cases. FIG. 53A-53B illustrates the automatic posting of a job. FIG. 54A-54C illustrates an overview of open, in progress, and completed jobs. FIG. 55A-55C illustrates that users can click on a completed job to review the work done and see an evaluation by the system if the submitted hours match up with the expected job. FIG. 56A-56B illustrates that users can view a replay of the actions that highlights the work completed. Submitting payment unlocks the hidden work

FIG. 57A-58 illustrates example interfaces for enabling the system to arbitrate when there is a dispute on how much work the expert performed. FIG. 57A-57B, If users want to dispute a payment they will see a summary of the hours worked and a verdict from the system on if the consultant has correctly charged them. FIG. 58A-58B illustrates that if the user wants to continue to dispute some of the time they can select the hours they feel they have been unfairly billed to begin the dispute process.

Executive View

FIGS. 59-62 illustrate some implementations of the current subject matter that can provide an executive overview of various AI and machine learning projects in a given enterprise showing how much value they are creating, how they are linked together, projects that could be creating value but haven't been deployed, and valuable insights from other analytical activities. FIG. 61A-61B illustrates a view in which a more optimal model for the enterprise has been identified and highlighted. FIG. 60A-60B illustrates a view in which a project that has not been integrated is highlighted and how much the expected impact of the integration will be is shown.

FIG. 70 is a system block diagram illustrating an example system 7000 that can enable collaboration of users, including subject matter expert and non-expert users, for developing and performing analytical tasks. The system 7000 includes a platform server 7005 that communicates with an enterprise system 7010 and a number of user devices 7015 via a network 7020. The enterprise system 7010 can include a database or dataset 7025 storing data relevant for performing analytical tasks. Each user device 7015 can include an input device and graphical user interface. The platform server 7005 can be capable of performing, for example, the process as described in FIG. 1 .

Although a few variations have been described in detail above, other modifications or additions are possible. For example, the above illustrated example represents an action graph utilizing hexagons and so can be considered to be restricted to the top 6 recommended steps at any point in the analysis. However, some implementation can be conducted using up to four recommended steps (squares), up to five (pentagons), up to eight (octagons), and the like. In some implementations, the underlying graph of granular analytical tasks that is used to enforce structure on the broader analytical task can be represented visually in many different ways. Similarly, while in this example color has been used to denote different X-axis variables in the data exploration examples (e.g., at FIGS. 16 and 17 ), shapes, icons and other visual representations can be used to denote such characteristics instead. While a clock-wise order or numbers have been used to indicate the most recommended next step, (or other information) alternative visual representations such as colors, icons, size, transparency levels, can be used to denote which is the most recommended next step, (or other information).

The subject matter described herein provides many technical advantages. For example, decomposing analytical tasks into assemblies of granular analytical tasks can enable the system to learn organizational best practices of various form and leverage such best practices to recommend next steps to users. For example, if users always filter out certain variables from sales data exported from a certain data source, the system can learn and automatically conduct such a step or at least recommend such a step to future users analyzing the same data. If users working on certain types of datasets tend to create certain derived variables, such as customer tenure in the case of customer churn analysis, the system can observe the granular analytics tasks of such users and recommend the addition of the appropriate derived variable when the dataset has the appropriate learned characteristics such as datasource, prior analysis by other users, variables, time since last access, and the like. If users analyzing marketing data typically draw and comment on a chart of marketing spend by region, then the system can start recommending such a chart whenever a user starts analyzing marketing data where the relevant variables exist for such a chart. By observing the types of granular analytical tasks performed by experts on different projects, the types of such projects, and the corresponding ratings or user acceptance rates of their work, the system can generate a granular evidence driven view of the nature of expertise of each expert, the kind of projects where they are expected to perform well and where they are not.

In some implementations, a similar analysis can be conducted for any kind of user to provide coaching recommendations based their success rates on different types of projects, the kinds of granular analytical tasks they tend to perform or not perform, the kinds of recommendations they tend to accept or ignore. For example, a user may be informed that they typically work on much smaller datasets than their peers working on similar projects and that their projects tend to be more popular or shared/used more broadly if they include at least 5 more fields or 20% more rows of data. Such analysis can also be used to better allocate analytical projects to the users most likely to succeed with such a project. Some implementations of the system can use request patterns for analytical projects to predict analyst demand for a specific time period and whether or not outsourced analysts may be needed to meet expected demand during the time period. Some implementations of the system can optimally allocate analytical tasks to the right analyst based on information about each analyst's project backlogs, expected demand, relative efficiency, and relative expertise for different types of analytical tasks.

One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) computer hardware, firmware, software, and/or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logical programming language, and/or in assembly/machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and/or device, such as for example magnetic discs, optical disks, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor. The machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example as would a processor cache or other random access memory associated with one or more physical processor cores.

To provide for interaction with a user, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as for example a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to the user and a keyboard and a pointing device, such as for example a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, such as for example visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including acoustic, speech, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices such as single or multi-point resistive or capacitive trackpads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

In the descriptions above and in the claims, phrases such as “at least one of” or “one or more of” may occur followed by a conjunctive list of elements or features. The term “and/or” may also occur in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it is used, such a phrase is intended to mean any of the listed elements or features individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases “at least one of A and B;” “one or more of A and B;” and “A and/or B” are each intended to mean “A alone, B alone, or A and B together.” A similar interpretation is also intended for lists including three or more items. For example, the phrases “at least one of A, B, and C;” “one or more of A, B, and C;” and “A, B, and/or C” are each intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” In addition, use of the term “based on,” above and in the claims is intended to mean, “based at least in part on,” such that an unrecited feature or element is also permissible.

The subject matter described herein can be embodied in systems, apparatus, methods, and/or articles depending on the desired configuration. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and/or variations can be provided in addition to those set forth herein. For example, the implementations described above can be directed to various combinations and subcombinations of the disclosed features and/or combinations and subcombinations of several further features disclosed above. In addition, the logic flows depicted in the accompanying figures and/or described herein do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations may be within the scope of the following claims. 

What is claimed is:
 1. A method comprising: receiving data characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs; calculating a first coefficient characterizing an interaction between a first input and a second input of the plurality of inputs based on a first set of weights of the plurality of weights, wherein the first set of weights are indicative of interaction between the first input and the second input; displaying, in a graphical user interface (GUI), a graphical object including a plurality of regions, wherein a first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively; displaying, in the GUI, a second graphical object connecting the first region to the second region, wherein a first visual characteristic of the second graphical object is indicative of the first coefficient.
 2. The method of claim 1, wherein the graphical object depicts a circle, wherein the first region and the second region are a first portion of the circumference of the circle and a second portion of the circumference of the circle.
 3. The method of claim 2, wherein the second graphical object depicts a chord connecting the first portion of the circumference of the circle to the second portion of the circumference of the circle, wherein the first visual characteristic is a thickness of the chord.
 4. The method of claim 1, wherein the impact function is a sum of a plurality of terms, wherein each term of the plurality of terms is a product of a weight of the plurality of weights and at least one input of the first plurality of inputs.
 5. The method of claim 4, wherein calculating the first coefficient is based on a first set of terms of the plurality of terms, wherein each term of the first set of terms is a product of the first input, the second input, and a weight of the first set of weights.
 6. The method of claim 5, wherein the first interaction coefficient is the sum of the first set of terms.
 7. The method of claim 4, further comprising: receiving data characterizing an analytical task and a target output of the analytical task; calculating, by the analytical task, the first set of weights based on the first plurality of inputs; and calculating an impact score by the impact function, wherein the impact score is indicative of a prediction of achieving the target output when inputs of the analytical task are the first plurality of inputs.
 8. A system comprising: at least one data processor; and at least one memory storing instructions which, when executed by the at least one data processor, result in operations comprising: receiving data characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs; calculating a first coefficient characterizing an interaction between a first input and a second input of the plurality of inputs based on a first set of weights of the plurality of weights, wherein the first set of weights are indicative of interaction between the first input and the second input; displaying, in a graphical user interface (GUI), a graphical object including a plurality of regions, wherein a first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively; displaying, in the GUI, a second graphical object connecting the first region to the second region, wherein a first visual characteristic of the second graphical object is indicative of the first coefficient.
 9. The system of claim 8, wherein the graphical object depicts a circle, wherein the first region and the second region are a first portion of the circumference of the circle and a second portion of the circumference of the circle.
 10. The system of claim 9, wherein the second graphical object depicts a chord connecting the first portion of the circumference of the circle to the second portion of the circumference of the circle, wherein the first visual characteristic is a thickness of the chord.
 11. The system of claim 8, wherein the impact function is a sum of a plurality of terms, wherein each term of the plurality of terms is a product of a weight of the plurality of weights and at least one input of the first plurality of inputs.
 12. The system of claim 11, wherein calculating the first coefficient is based on a first set of terms of the plurality of terms, wherein each term of the first set of terms is a product of the first input, the second input, and a weight of the first set of weights.
 13. The system of claim 12, wherein the first interaction coefficient is the sum of the first set of terms.
 14. The system of claim 11, the operations further comprising: receiving data characterizing an analytical task and a target output of the analytical task; calculating, by the analytical task, the first set of weights based on the first plurality of inputs; and calculating an impact score by the impact function, wherein the impact score is indicative of a prediction of achieving the target output when inputs of the analytical task are the first plurality of inputs.
 15. A non-transitory computer readable medium storing executable instructions that, when executed by at least one processor forming part of at least one computing system, cause the at least one processor to perform operations comprising: receiving data characterizing an impact function, a first plurality of inputs to the impact function, and a plurality of weights associated with the first plurality of inputs; calculating a first coefficient characterizing an interaction between a first input and a second input of the plurality of inputs based on a first set of weights of the plurality of weights, wherein the first set of weights are indicative of interaction between the first input and the second input; displaying, in a graphical user interface (GUI), a graphical object including a plurality of regions, wherein a first region and a second region of the plurality of regions are indicative of the first input and the second input, respectively; displaying, in the GUI, a second graphical object connecting the first region to the second region, wherein a first visual characteristic of the second graphical object is indicative of the first coefficient.
 16. The non-transitory computer readable medium of claim 15, wherein the graphical object depicts a circle, wherein the first region and the second region are a first portion of the circumference of the circle and a second portion of the circumference of the circle.
 17. The non-transitory computer readable medium of claim 16, wherein the second graphical object depicts a chord connecting the first portion of the circumference of the circle to the second portion of the circumference of the circle, wherein the first visual characteristic is a thickness of the chord.
 18. The non-transitory computer readable medium of claim 15, wherein the impact function is a sum of a plurality of terms, wherein each term of the plurality of terms is a product of a weight of the plurality of weights and at least one input of the first plurality of inputs.
 19. The non-transitory computer readable medium of claim 18, wherein calculating the first coefficient is based on a first set of terms of the plurality of terms, wherein each term of the first set of terms is a product of the first input, the second input, and a weight of the first set of weights.
 20. The non-transitory computer readable medium of claim 19, wherein the first interaction coefficient is the sum of the first set of terms. 